Method and system for adaptive perspective correction of ultra wide-angle lens images

ABSTRACT

Methods for correcting an image acquired through an Ultra Wide-Angle (UWA) lens is disclosed, where the entire image is mapped via a transformation that substantially improves the perspective, yet maintains all image information. Additional locally adaptive transformations can be applied to regions of interest in the image to fully restore the perspective. This de-warping scheme allows for flexible manipulation of the image while no content would be lost.

FIELD

The embodiments described herein relate generally to electronic imageand video processing, and particularly to correcting images severelydistorted by wide-angle lenses.

BACKGROUND

Wide-angle lenses are commonly used in cameras amongst other image andvideo capture devices where a large viewing angle is required orpreferred. There also exist so-called Ultra Wide-Angle (UWA) lenses,with Field of Views (FOV) of up to or even larger than 180°. A fisheyelens is an exemplary UWA that may cover up to 180° (2π steradians) ofangular FOV. As such, having two fisheye lenses sitting back to back andfacing opposite directions will in principle cover a 360° view of thesurroundings.

Image capture systems equipped with UWA lenses have many usefulapplications. Examples are surveillance and security monitoring cameras,video-conferencing systems, specialty photographic cameras, and vehiclecamera systems. On a negative side, these lenses exhibit significantamount of distortions that pose a number of challenges. For example, theoutput of a typical fisheye lens is normally a two-dimensional (2D)circular (oval for a general wide-angle lens) mapping of a 3D visuallyfamiliar Euclidian space where objects lie in real world. Inevitably,horizontal and vertical lines would look skewed and appear curvilinear,more noticeably for the objects farther away from the center of thelens. The relative sizes of objects closer to the lens lookexaggeratingly larger, so that different parts of an object look out ofproportion. These effects are generally known as the fisheye distortionsthat make it difficult to judge distances and object relationships. Thisis in contrast with standard camera lenses which maintain the sceneperspective and linearity by capturing a planar view. Moreover, toachieve a larger angular FOV, manufactures may use a combination ofseveral lenses. That may further introduce optical and geometricdistortions.

Several techniques are known in the art to address the fisheyedistortions. The curvilinear images produced by a fisheye lens can beremapped onto a conventional rectilinear display. In a series of priorart patents from OmniView and IPIX Corp. a portion of the full viewingsphere is selected and corrected for fisheye distortions. Although thisapproach provides a powerful tool for viewing up to 360°, only aselected portion is available at a time. The rest of the image is savedthat could be available upon further selections.

U.S. Pat. No. 7,961,980B2 utilizes cylindrical projections in connectionwith circular fisheye mappings. As is well known, cylindricalprojections stretch distances east-west. In particular, the stretchingdiverges to infinity at the poles. This solution would be useful forcreating panoramic images when the polar areas can be chopped off, againmeaning loss of information. Although it is possible to take differentorientation cylinders to build several images, still this does notprovide a single zero-content-loss (ZCL) image, that is, an imagecontaining all content from the original UWA captured scene, with noloss of information. In fact, it is physically impossible to achievefull perspective correction for all portions of the image at the sametime by applying any transformation.

Therefore, the main drawback of prior art is loss of image content andinformation pertaining to objects and their relative positioning. Incertain applications, such as a security or monitoring camera, it wouldbe desired to be able to select areas of interest and correct fordistortions, and yet be able to preserve the full content of the imageat all times. The present invention discloses an intermediate approach,whereby image data for all pixels of the entire UWA lens captured imageis maintained to prevent any loss of information after the mapping thatcorrects for distortions to a certain level, as much as theoreticallyallowed. Furthermore, a locally adaptive transformation can be appliedto the regions of interest that aims to partially restore thedistortions caused by the UWA lens. Partial image restoration has theobjective of giving a reasonable indication of the object and itsrelationships to the surroundings, not a strict full-image perspectivecorrection. This solution will be particularly useful for videomonitoring applications where a full view of the surveillance must beavailable for every image frame, yet it would be possible to transformobjects of interest.

SUMMARY

The embodiments described herein provide in one aspect, a method fortransforming an input image captured through at least one ultrawide-angle (UWA) lens, said UWA lens characterized by a correspondingtransformation that maps 3D object space onto a plane that displays theinput image, said method comprising:

-   -   (a) obtaining the input image data captured through the at least        one UWA lens;    -   (b) constructing a 2D surface envelope that fully encompasses        field of view of the at least one UWA lens;    -   (c) constructing an envelope transformation that maps an output        image plane onto the surface envelope;    -   (d) concatenating the UWA lens map and the envelope        transformation to obtain a zero-content-loss transformation; and    -   (e) applying the zero-content-loss transformation to the input        image data to obtain an output image;        such that image perspective is substantially improved in the        output image.

The embodiments described herein provide in another aspect an electronicprocessor for transforming an image captured through at least one ultrawide-angle (UWA) lens, said UWA lens characterized by a correspondingtransformation that maps 3D object space onto a plane that displays theinput image said processor comprising:

-   -   (a) means to obtain input image data captured through the at        least one UWA lens;    -   (b) means to select a 2D surface envelope that contains field of        view of the at least one UWA lens;    -   (c) means to construct an envelope transformation that maps an        output image plane onto the surface envelope;    -   (d) means to concatenate the UWA lens map and the envelope        transformation to obtain a zero-content-loss transformation; and    -   (b) means to apply the zero-content-loss transformation to the        input image data to obtain an output image;        such that image perspective is substantially improved in the        output image.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the embodiments and/or relatedimplementations described herein and to show more clearly how they maybe carried into effect, reference will now be made, by way of exampleonly, to the accompanying drawings which show at least one exemplaryembodiment and/or related implementation in which:

FIG. 1 illustrates an embodied method for applying a zero-content-losstransformation to an UWA lens acquired image;

FIG. 2 illustrates the conventional Cartesian and spherical coordinatesystems and conventions used in the disclosure;

FIGS. 3a and 3b illustrate an exemplary spherical and ellipsoidFOV-envelope respectively for a 180° UWA lens;

FIGS. 4a and 4b illustrate an exemplary spherical and ellipsoidFOV-envelope respectively for a 270° UWA lens;

FIG. 5a illustrates a cylindrical mapping using a hemi-sphericalFOV-envelope for a number of grid points, and FIG. 5b shows thecorresponding grid as it would appear on the input image;

FIG. 6 illustrates an exemplary input image acquired through a 180° UWAlens as part of a monitoring device;

FIG. 7 illustrates an exemplary input image acquired through a 270° UWAlens as part of a monitoring device;

FIG. 8 illustrates a result of a zero-content-loss transformationapplied to the input image of FIG. 6 using Lambert cylindricalprojection;

FIG. 9 illustrates a result of a zero-content-loss transformationapplied to the input image of FIG. 6 using Zenith linear cylindricalmapping;

FIG. 10 illustrates a result of a partial zero-content-losstransformation applied to the input image of FIG. 6 using Mercatorcylindrical mapping;

FIG. 11 illustrates an ellipsoid envelope covering a FOV >180°;

FIG. 12 illustrates mapping the boundary of the FOV-envelope;

FIG. 13 illustrates mapping of the center column of pixels from theoutput image plane to the FOV-envelope;

FIG. 14 illustrates a plane uniquely defined by mapping of two boundarypoints and one inside point from a row of the output image;

FIG. 15 illustrates an exemplary envelope transformation for a 270° UWAlens for a set of points on the output image;

FIG. 16 illustrates an exemplary envelope transformation for a 180° UWAlens for a set of points on the output image;

FIG. 17 illustrates a result of a zero-content-loss transformationapplied to the input image of FIG. 6 using an exemplary ZCL mapping;

FIG. 18 illustrates a result of a zero-content-loss transformationapplied to the input image of FIG. 7 using an exemplary ZCL mapping;

FIG. 19 illustrates a spherical envelope with the overlaying envelopemapping for a 270° UWA lens;

FIG. 20 illustrates a simulated image from a UWA lens wherein thecaptured input image is an ellipse;

FIG. 21a illustrates an ZCL envelope mapping for a 180° UWA lens, and aperspective envelope mapping for a specific region of interest;

FIG. 21b illustrates the stitched envelope mapping, combining the ZCLand perspective envelope mappings of FIG. 21 a.

FIG. 22 is essentially the same input image as in FIG. 6 with twoselected regions of interest in the image.

FIG. 23 illustrates the result of the zero-content-loss transformationessentially as in FIG. 17 showing two selected regions of interest;

FIG. 24 illustrates the result of a local perspective transformation,within a ZCL transformation, adapted to correct the lower region ofinterest;

FIG. 25 illustrates the result of a local perspective transformation,within a ZCL transformation, adapted to correct the upper left region ofinterest, in addition to the lower right region of interest;

FIG. 26a illustrates an exemplary large region of interest (FOV of 90°),selected as the middle room in the image, that is corrected forperspective;

FIG. 26b illustrates the result of a local perspective transformationadapted to correct the perspective and zoom the large selected area;

FIG. 26c illustrates the result of a local perspective transformationadapted to correct for the perspective for the two side rooms;

FIG. 26d illustrates the result of a combination of the two previousperspective corrected areas;

FIG. 27 illustrates the result of four locally perspective correctedrooms, each corresponding to a 90° FOV as extracted from a 360° view;

FIG. 28 illustrates an exemplary input image where areas of interestedare selected in response to activity;

FIG. 29 illustrates the result of local perspective transformationsadapted to correct for perspective (within a ZCL mapping), in additionto zoom and rotation transformations, in order to prepare the image forfurther video analytics.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION

It will be appreciated that numerous specific details are set forth inorder to provide a thorough understanding of the exemplary embodimentsdescribed herein.

However, it will be understood by those of ordinary skill in the artthat the embodiments and/or implementations described herein may bepracticed without these specific details. In other instances, well-knownmethods, procedures and components have not been described in detail soas not to obscure the embodiments and/or implementations describedherein. Furthermore, this description is not to be considered aslimiting the scope of the embodiments described herein, but rather todescribe the structure and operation of the various embodiments and/orimplementations described herein.

It is well understood that an image acquired through a UWA lens cannotbe unfolded on a normal (usually rectangular) display in its entirety,while having it perfectly corrected for perspective distortions. Asmentioned, the prior art solutions have to sacrifice portions of theimage content, perhaps deemed as less important, in order to correct forthe rest. In accordance with an embodiment of the present invention, anintermediate solution is provided where the entire UWA lens acquiredimage is unfolded via a transformation that provides a high degree ofcorrection, yet maintains all image information. A localized and contentadaptive correction can further be applied to the key regions ofinterest, if required, which aims to partially restore some of theperspective lost or severely distorted by the UWA lens. Such a full-viewtransformation, with possible locally adaptive perspective corrections,will be referred to as Zero-Content-Loss (ZCL) correction. Partialrestoration aims to give a reasonable indication of objects and theirrelationships, and not a strict perspective correction where loss ofcontent would be unavoidable. The restoration process can also be viewedas redistributing the original distortions to regions of lessimportance.

To describe the embodiments, mainly an exemplary 180° lens will be usedas the UWA lens to describe the method. Although it should be understoodthat this approach is applicable generally to any UWA lens or acombination of lenses covering up to a 360° of field of view. Inparticular, the embodied methods are applicable to custom UWA lenseswhich do not necessarily image as a circle, but may rather image as anellipse or oval, such as the Panomorph lens from ImmerVision.

FIG. 1 illustrates an embodied method according which azero-content-loss transformation is implemented. An image is captured100 by a capture device equipped with a UWA lens, the captured image isreferred to as the input image. Digital capture devices are normallyequipped also with an image sensor, such as a CCD or CMOS, through whichone can obtain input image data 110 comprising spatial and colorcoordinates of every and each image pixel. In absence of any perspectivecorrections, the input image is usually viewed as within an oval(circular being a special case) image boundary. This is referred to asthe lens mapping, whereby a view from the real world three-dimensional(3D) object space is mapped onto a planer two-dimensional (2D) planeview. The lens mapping transformation needs to be identified 120 eitheras provided by the lens manufacturer, or modeled mathematically.Alternatively, a more precise and general mapping may be obtainedempirically. Next, a 2D surface is selected 130 to fully encompass theFOV of the lens in 3D. This covering surface also referred to as theFOV-envelope, will later be described in great detail. A transformation140 is then constructed to map an output image plane onto theFOV-envelope surface, where every pixel is accounted for, and in such away that the output image 170 is substantially improved for perspective.The output image, displayed on a viewing surface, is thereforetransformed in such a way that the perspective distortions are improvedwhile all image content is preserved (hence ZCL transformation). That isachieved by concatenating the lens mapping and the FOV-envelopetransformation 150, and applying the result to the input image 160.

Selecting the FOV-envelope and mapping of the output image to thesurface of the FOV-envelope determine the amount and areas of theperspective transformations. Therefore if one requires local perspectiveadjustments to the transformations 180, one could re-select the envelope130 and start over in order to modify the displayed output 170accordingly. Alternatively, a different envelope transformation 140 canbe applied with the same envelope (no need to alter 130) to tweak theoutput image. The local adjustments can be performed manually via anoperator by selecting one or more areas of interest that need to becorrected, or it could be automated in response to changes in thecaptured image. For instance, a security camera might be equipped with amotion detector. Once a movement is detected, areas around the movingobject (e.g. face of a person) can be selected as an area of interestfor the local adjustments 180. Additional processing could also beapplied to the selected areas, such as zooming in/out, rotation andreflection operations. This would assist image analytic techniques forpost-processing, e.g. face recognition.

The imaging of a standard camera (SLR type), with a standard non-UWAlens (example 35 mm lens) can be described using the ideal pinholecamera model. Though real cameras are much more complicated, this simplemodel suffices for the description below. A point in the 3D real worldspace with coordinates (x,y,z) is mapped to a point in the image (u,v)given by:

$\begin{matrix}{{u = {f\frac{x}{z}}},{v = {f\frac{y}{z}}}} & (1)\end{matrix}$where f is the lens focal length and the optical axis is assumed to bealong the z-axis. Equation (1) is referred to as the lens mappingequation for a standard lens; it maps a real world point to a point inthe captured image. The image can be viewed as placed at a distance ffrom the lens center along the optical axis. The standard lenstransformation preserves relative distances and angles of objects thatare in the same z-plane, i.e. a plane perpendicular to the z-axis atsome z coordinate, hence perspective is maintained. The resulting imageis rectangular. Furthermore, rectangular coordinates are a naturalcoordinate system for describing non-UWA lens imaging. In contrast, aUWA lens is best described using a spherical/polar coordinate system.Let be (r,θ,φ) the spherical coordinates of a point in the 3D real worldspace (its rectangular coordinates being (x,y,z)) as shown in FIG. 2. AUWA lens 200 placed at the origin and looking up in +z direction mapsthe 3D point to a point (r′,φ′) in the image plane (not shown) given by:r′=F _(r) ^(L)(θ,φ),φ′=F _(θ) ^(L)(θ,φ)  (2)where the superscript L refers to “Lens” and (r′,φ′) are the polarcoordinates in the image related to rectangular coordinates by: x′=r′cos φ′ and y′=r′ sin φ′. In terms of rectangular coordinates (2) can bewritten as:

$\begin{matrix}{{{u = {{F_{u}^{L}\left( {x,y,z} \right)} = {{F_{r}^{L}\left( {\theta,\phi} \right)}{\cos\left( {F_{\theta}^{L}\left( {\theta,\phi} \right)} \right)}}}},{v = {{F_{v}^{L}\left( {x,y,z} \right)} = {{F_{r}^{L}\left( {\theta,\phi} \right)}{\sin\left( {F_{\theta}^{L}\left( {\theta,\phi} \right)} \right)}}}}}{{{\theta\left( {x,y,z} \right)} = {\arccos\frac{z}{\sqrt{x^{2} + y^{2} + z^{2}}}}},{{\phi\left( {x,y,z} \right)} = {\arctan\left( \frac{y}{x} \right)}}}} & (3)\end{matrix}$It is understood that the inverse trig operations need to take intoaccount quadrants and handle x=0. The functions are general with nodependence on the radial coordinate r; in particular they are alsoapplicable to lenses which form elliptical or oval images. UWA lenseswith no radial dependence will be referred to as conical-UWA lens. Forspecial case of an ideal fisheye, these equations simplify to:r′=αθ,φ′=φ, αa constant  (4)Equations (2) to (4) are referred to as the conical-UWA lens and fisheyelens mapping equations respectively. Imaging obeying Equation (2) or (4)does not maintain relative distances and angles on fixed planes, leadingto much distorted images. For example, for Equation (4), all points on acone at a fixed θ map onto a circle in the image plane. The resultingfisheye image is circular as opposed to rectangular.

The benefit of the UWA lens, offsetting the severe distortion, is itsmuch larger FOV. A fisheye field-of-view is defined as twice the maximumangle θ that can be captured, which gives a cone from the lens center:Fisheye FOV=2θ_(max)  (5)For a 180° fisheye θ_(max)=90, where the lens can capture a fullhemispherical field of view. A standard lens, even with a wide anglevariation is usually restricted to <=120° horizontally and much smallervertically. We define a conical-UWA lens FOV by:

$\begin{matrix}{{{Conical} - {{UWA}\mspace{14mu}{FOV}}} = {2 \times {\min\limits_{\phi}\left\{ {\theta_{\max}(\phi)} \right\}}}} & (6)\end{matrix}$This definition implies that θ_(max) may depend on φ, and the minimum istaken. For most practical cases, θ_(max) is independent of φ andequation (5) can also be taken as the conical-UWA lens FOV.

Understanding the FOV is critical to implementing a zero-content-losscorrection. To this extent we define a FOV-envelope for a lens, asmentioned in relation to coordinate system of FIG. 2, as a 2D surfacecovering (embedded in real world 3D space) that fully encompasses thefield of view of the lens. Every ray drawn from an object point in thereal world (object space) to the lens center, which is within the lensfield of view, intersects the FOV-envelope at a unique point.Conversely, every point on the envelope corresponds to a unique ray thatis captured by the lens. The FOV-envelope would not be unique. Ingeneral there are an infinite number of such surfaces for a given lens,however there will be a basic envelope which is the smallest and has asimple mathematical description. All other envelopes will be continuoustransformations of the basic envelope. Two envelopes are considered asbeing equivalent if they can be continuously transformed into eachother; an envelope equivalent to the basic envelope is also referred toas being an equivalent. The standard spherical and Cartesian coordinatesof the envelope are denoted by (r,θ,φ) and (x,y,z) respectively, whichis the same notation used for the 3D real world space as in FIG. 2. Forstandard lenses (Equation (1)), the basic FOV-envelope would be a planarrectangular region containing the FOV, perpendicular to the opticalaxis. This is identical to the image plane at focal length f. For aconical-UWA lens, and in particular a fisheye lens, the basicFOV-envelope can be taken as the surface of a sphere of unit radius,with θ≤θ_(max):Conical-UWA FOV-envelope={r=1,0≤θ≤θ_(max),0≤φ<2π}  (7)Every point on the FOV-envelope, which corresponds to a real world ray,has its position in the captured image determined by the lens mappingequations. Given an envelope, an envelope mapping (or transformation)140 can be defined, which combined with the lens mapping 120 results ina ZCL transformation 150. This is described in the following,referencing FIG. 1.

Let (u,v) be the coordinates of the 2D input image data 110 captured bythe UWA lens. Since image sensors (used in digital cameras, etc.) arerectangular, this will be a rectangular frame with pixel resolution(width×height) W₀×H₀, containing within it the oval or circular UWAimage. Let (x_(f), y_(f)) be the coordinates of the 2D image after thedistortion correction, where the output image 170 is assumed to be arectangular image of pixel resolution W₁×H₁. In most cases, the inputand output resolutions are the same: W₁×H₁=W₀×H₀. However, the notationis kept general to allow for a change in resolution as well. An envelopemapping 140 is defined as a mapping (F_(x) ^(C),F_(y) ^(C),F_(z) ^(C))of the output coordinates (x_(f), y_(f)) onto a basic (or anyequivalent) FOV-envelope of the lens ((x,y,z) denote coordinates on theenvelope):x=F _(x) ^(C)(x _(f) ,y _(f))y=F _(y) ^(C)(x _(f) ,y _(f))z=F _(z) ^(C)(x _(f) ,y _(f))  (8)An equivalent set of spherical coordinate equations can also be written:r=F _(r) ^(C)(x _(f) ,y _(f))θ=F _(θ) ^(C)(x _(f) ,y _(f))φ=F _(φ) ^(C)(x _(f) ,y _(f))  (9)The two sets related in the usual manner:

$\begin{matrix}{{F_{r}^{C} = \sqrt{\left( F_{x}^{C} \right)^{2} + \left( F_{y}^{C} \right)^{2} + \left( F_{z}^{C} \right)^{2}}}{{F_{\theta}^{C} = {\arccos\frac{F_{z}^{C}}{\sqrt{\left( F_{x}^{C} \right)^{2} + \left( F_{y}^{C} \right)^{2} + \left( F_{z}^{C} \right)^{2}}}}},{F_{\phi}^{C} = {\arctan\left( \frac{F_{y}^{C}}{F_{x}^{C}} \right)}}}} & (10)\end{matrix}$

Maps (8) and (9) are also referred to as the envelope covering; hencethe superscript “C” is used herein. Every 3D point on the FOV-envelopehas one point in the output 2D image which maps onto it. Further, everypoint on the envelope will have a unique mapped point in (x_(f),y_(f)).By definition of the FOV-envelope, this ensures that every point seen inthe captured image will also be seen in the output image hence providingno loss of content. A zero-content-loss (ZCL) transformation 150 isobtained by concatenating the envelope mapping 140 with the lens mapping120. This provides a mapping between the output and the input imagecontaining all the original content, which is used to determine thepixel value at (x_(f),y_(f)). The equations for a ZCL transformationtake the form:u=F _(u) ^(L)(F _(x) ^(C)(x _(f) ,y _(f)),F _(y) ^(C)(x _(f) ,y _(f)),F_(z) ^(C)(x _(f) ,y _(f)))≡F _(u) ^(ZCL)(x _(f) ,y _(f))v=F _(v) ^(L)(F _(x) ^(C)(x _(f) ,y _(f)),F _(y) ^(C)(x _(f) ,y _(f)),F_(z) ^(C)(x _(f) ,y _(f)))≡F _(v) ^(ZCL)(x _(f) ,y _(f))  (11)Analogous equations can be written in spherical/polar coordinates.

The zero-content-loss transformation is designed to restore a certainlevel of perspective in areas of interest. Therefore it is also referredto as a zero-content-loss correction. As mentioned above, this is notstrict perspective correction (which is theoretically impossible for thefull view), but an approximation to provide a better viewing experienceand an improved sense of spatial placement of objects. The aboveframework provides with two means of modifying the correction:

-   -   a) Selecting a different basic equivalent FOV-envelope 130, via        a continuous transformation of the starting envelope.    -   b) Modifying the envelope transform 140. Note that the lens        mapping 120 is determined by the lens and is not an adjustable        transform.        It should further be noted that a) also includes a change in the        envelope transformation, however b) may be implemented without        a). By appropriately modifying the envelope and its transform,        different types of correction can be obtained.

According to one embodiment, the ZCL correction in the present approachhas the ability to adapt to the image content for further correction oflocal perspective and other transformations in a specific region. Inwhat follows, a systematic approach to constructing local perspectiveadjustments will be described. A correction with this property will bereferred to as being locally-perspective-adaptive (LPA). Preferably, theFOV-envelope and the envelope transform are built in a parameterizedmanner, so that they can be quickly modified by changing specificparameters. This can often be achieved by starting with an envelope andtransformation, whose mathematical properties are simple to handle. Oncea ZCL correction is obtained either in equation form or as a set of gridpoints, it can be applied to images or video signals at real-time rates.For example, a highly efficient image warping hardware as described inthe U.S. Pat. No. 7,324,706 can be employed for achieving this.

To illustrate the embodied method, a 180° conical-UWA lens is describedas an exemplary UWA lens, with the lens mapping simplified byrestricting the conical-UWA lens to a fisheye lens. For a 180°conical-UWA lens, the basic FOV-envelope with θ_(max)=90 is a halfsphere, as shown in FIG. 3a . FIG. 3a is consistent with the coordinateconvention introduced in FIG. 2, i.e. the lens is placed at the origin(0, 0, 0) and looks outward along the +z axis (the optical axis). Theangle θ is measured from the +z axis, and the angle φ is measuredclockwise from the +x axis. An equivalent FOV-envelope could be a halfellipsoid, shown in FIG. 3b with the y semi-axis being longer. Any halfellipsoid with arbitrary semi-axes lengths forms an equivalentFOV-envelope. The ellipsoid, that in fact generalizes a sphere, willprovide a different ZCL correction in comparison with the sphere. Inanother example, FIGS. 4a and 4b show two equivalent FOV-envelopes (asphere and an ellipsoid) for UWA lenses with >180° FOV, specifically270° as depicted.

Having identified a FOV-envelope, the next step is to construct theenvelope transformation 140, which is the mapping of the output imageonto the envelope. In an exemplary illustration within the framework ofZCL, a good starting point would be to consider standard projectionsonto the globe used by cartographers to build flat 2D maps. Inparticular, the most relevant is the cylindrical projection, whosemathematics is readily understandable. The polar divergence trouble ofcylindrical transformations, as described above, makes cylindricalprojections less useful for constructing ZCL transformations. Regardlessof the limitations, it is informative to see the equations for a ZCLtransformation built via cylindrical projection.

There exist various types of cylindrical projections. Lambert projectionis one example, employed here without loss of generality. With referenceto FIG. 3a , the cylinder would have its axis along the y axis and isformed by wrapping the output image (rectangular of resolution W₁×H₁)around the basic envelope in FIG. 3a . The poles are at(r=1,θ=π/2,φ=π/2) and (r=1,θ=π/2,φ=−π/2). The cylindrical mapping fromthe image coordinates (x_(f),y_(f)) to the FOV-envelope is given by:

$\begin{matrix}{{\phi_{xz} = {{\pi\left( {1 - \frac{x_{f}}{W_{1}}} \right)} \equiv {\pi\;{\overset{\_}{x}}_{f}}}}{{\cos\;\theta_{y}} = {\left( {1 - {2\frac{y_{f}}{H_{1}}}} \right) \equiv {\overset{\_}{y}}_{f}}}} & (12)\end{matrix}$where the range of (x_(f),y_(f)) is [0,W₁]×[0,H₁], i.e. the outputimage, θ_(y) is the polar (zenith) angle relative to the +y axis andφ_(xz) is the azimuthal angle in the xz-plane measured from the +x axis,with clockwise towards the +z axis. The choice of using (θ_(y),φ_(xz))is dictated by the orientation of the cylinder relative to the envelope.This can be converted to the standard coordinates using (even though r=1on the basic envelope its dependence via r=R is explicitly shown, whichwill be useful when modifying envelopes):y=Rcos θ_(y)R _(xz)≡√{square root over (R ² −y ²)}x=R_(xz) cos φ_(xz)=R sin θ_(y) cos φ_(xz)z=R_(xz) sin φ_(xz)=R sin θ_(y) sin φ_(xz)  (13)Converting to the standard spherical coordinates:

$\begin{matrix}{{r = R}{\theta = {\arccos\left( \frac{z}{R} \right)}}{\phi = {\arctan\left( \frac{y}{x} \right)}}} & (14)\end{matrix}$It should be understood that the arc-tangent needs to take into accountquadrants and handle x=0. Equations (12), (13) and (14), all give thesame envelope transform based on cylindrical projection. Thistransformation is illustrated in FIG. 5a for a given set of points onthe output image. To compute the final zero-content-loss correction, wenext map the envelope coordinates onto the input image using the mappingequations (3) and (4). For the 180° fisheye lens, combining (13) and (4)gives:

$\begin{matrix}{{u = {\alpha\;{\arccos\left( {\sin\;\theta_{y}\sin\;\phi_{xz}} \right)}\frac{\sin\;\theta_{y}\cos\;\phi_{xz}}{\sqrt{{\cos^{2}\theta_{y}} + {\sin^{2}\theta_{y}\cos^{2}\phi_{xz}}}}}}{v = {\alpha\;{\arccos\left( {\sin\;\theta_{y}\sin\;\phi_{xz}} \right)}\frac{\cos\;\theta_{y}}{\sqrt{{\cos^{2}\theta_{y}} + {\sin^{2}\theta_{y}\cos^{2}\phi_{xz}}}}}}{\alpha = \frac{R_{I}}{\theta_{\max}}}} & (15)\end{matrix}$where R_(I) is the radius of the circular fisheye input image capturedby the lens; the equation for the proportionality factor α is that of acircular fisheye.

Referring to FIG. 5b , the coordinate system on the input image space istaken to be the standard Cartesian coordinates, with the origin at thecenter, +u to the right and +v to the top. This can be converted toconventional computer coordinates. From equation (15) one can see thatthe lines y_(f)=0 and y_(f)=H₁, corresponding to θ_(y)=0 and θ_(y)=πrespectively, are mapped to a single point in the input image:(u=0,v=+R_(I)) and (u=0,v=−R_(I)). This mapping completely destroys theperspective towards the top and bottom of the output image; hence thisis not a very useful ZCL correction. FIG. 5a illustrates the enveloptransform and FIG. 5b illustrates the final ZCL transformation for a setof gird points on the output image, using the cylindrical projection.

By modifying the envelop transform (12), within the framework ofcylindrical projections, variations of the ZCL transformation can beobtained. However, the same “infinite stretching” limitation isencountered for all such cases. Two exemplary variations, which amountto modifying the second relation in (12), are given below:

$\begin{matrix}\begin{matrix}{\theta_{y} = {\pi\;\frac{y_{f}}{H_{1}}}} & \left( {{Zenith}\mspace{14mu}{Linear}} \right) \\{\theta_{y} = {\frac{\pi}{2} - {\arctan\left( {\sinh\left( {s_{f}\left( {1 - {2\frac{y_{f}}{H_{1}}}} \right)} \right)} \right)}}} & ({Mercator})\end{matrix} & (16)\end{matrix}$

Equation (15) continues to hold since it is based on the lens mappingequation and does not change. Combining (15) with the new substitutions(16), the zero-content-loss transformation can be computed. The firstvariation, which is labeled zenith linear, changes the scale in the ydirection in comparison with the Lambert projection; the same infinitestretching is still seen at the poles. The second variation is actuallya Mercator cylindrical projection, where s_(f) is a scale factor thatcontrols the vertical stretching. This becomes a zero-content-losstransformation only in the limit s_(f)→∞, in which case there isinfinite vertical stretching. Nevertheless, even for small (s_(f)≥4),which may be “zero-content-loss” for most practical purposes, thedistortion introduced is very severe.

Referring to FIG. 6, a simulated 180° conical-UWA (circular) lens imageis used as the input image to illustrate how an exemplaryzero-content-loss transformation performs, in accordance with theembodiments of the invention. The scene depicted in FIG. 6 consists ofthree similar size and adjacent rooms, wherein rooms 620 and 640 are atright angle to 600. In other words, the room fronts form 3 interiorfaces of a cube. A UWA lens placed at the center of the cube is lookingdirectly (and along the perpendicular bisectors) at the middle room 600.The distortions introduced by the lens, increasing in the radialdirection from the center, are clearly seen. Towards the image edge,objects become more “curved”. Nevertheless being a UWA lens, it doescapture everything in a 180° hemispherical view including parts of theceiling 660 and the floor 680. The setting of FIG. 6 can easily beextended to a 360° view by adding a second similar lens facing theopposite direction. Therefore, what is said in regard to one fisheyelens can be extended to two fisheyes. Another simulated image shown inFIG. 7 illustrates input image acquired from a 270° lens, wherein theentire side rooms 720 and 740 (show on image) are within the FOV,although severely distorted as expected. Both FIGS. 6 and 7 are formedby a lens obeying lens mapping equation (4).

FIGS. 8 to 10 show the result of ZCL transformations, based on thecylindrical projection (15) applied to the input image FIG. 6. Teachingsof the U.S. Pat. No. 7,324,706 have been employed to obtain output FIGS.8 to 10 by implementing the given transformations in it's de-warpscheme. FIG. 8 is based on the Lambert projection, FIG. 9 on the zenithlinear projection and FIG. 10 on the Mercator projection (s_(f)=2) ofequation (16). The infinite stretching at the poles is apparent; this isespecially visible on the floor where the carpet at the center appearstorn towards opposite directions. In other words, all perspective nearthe poles is lost. The horizontal object lines are also made more curvedthan the original UWA lens image which distorts the perspective towardsthe left and right of the field of view. For instance, this mappinggives the illusion that the three rooms all lie along a line. Further,the diagonal ceiling/floor boundaries of the left and right rooms becomefully horizontal at the edges, whereas a less distorted perspectivewould keep these lines diagonal. However, as needed for perspective, thevertical object lines are straightened. These properties makecylindrical projections not well suited for ZCL transformations aimingto improve perspective. Nevertheless, the straightening out of verticalsand diagonals at edges does make these transform useful for panoramicviews, where the zenith angle is usually cropped to a smaller range.

According to a preferred embodiment, a different envelope transform canbe constructed to avoid the above artifacts resulted from cylindricalprojections. This approach is specifically tailored towards ZCLtransformation with perspective correction, using a parameterizedenvelope transform that can be modified to adjust the regions of, andamounts of, residual distortion. Furthermore, it includes a correctionmechanism for recovering true local perspective. This technique supportsany UWA lens, including non-circular conical-UWA lenses. Supportingarbitrary lenses in the context of ZCL correction is a directconsequence of the embodied method (as described in reference to FIG.1), whereby an envelope mapping is defined separate from the lensmapping.

Accordingly, consider a conical-UWA lens with basic envelope that maycover greater than 180° FOV and is not limited to capturing circularimages. An equivalent envelope is given by the partial surface of anellipsoid shown in FIG. 11:

$\begin{matrix}{{{\frac{x^{2}}{r_{x}^{2}} + \frac{y^{2}}{r_{y}^{2}} + \frac{z^{2}}{r_{z}^{2}}} = 1}{\theta \leq \theta_{\max}}} & (17)\end{matrix}$The envelope 1100 is the part of the ellipsoid that lies above θ_(max).Here r_(x,y,z) denote the semi-axes lengths. The ellipsoid generalizes asphere and provides three parameters, the semi-axes lengths, formodifying the envelope transform. The lens is again assumed to besitting at the origin looking towards the +z axis. r_(x) and r_(y)provide scaling controls in the x and y directions respectively, whereasr_(z) gives control of scaling in the z direction corresponding to amagnification. The ellipsoid envelope 1100 of FIG. 11 is drawn for ageneral value of the semi-axes. Accordingly, the boundary θ=θ_(max) isan elliptical shape 1150. Note the boundary in general does not lie in aplane for an ellipsoid. The boundary 1150 will be referred to as theenvelope boundary or e-boundary.

Referring to FIG. 11, the envelope transformation maps an outputrectangular image W₁×H₁ (not shown) onto the ellipsoid surface 1100. Inthe first step of the construction, the image rectangular boundary ismapped to the boundary 1150 of the envelope. Looking along the z-axis,the boundary 1150 appears as an ellipse (for a sphere it becomes acircle); as illustrated 1250 in FIG. 12. We define an angle 0<ψ₀<π/2,called the boundary partition angle, which divides the boundary into 4segments (note θ=θ_(max) for the segments):

$\begin{matrix}{{{{{Top}\mspace{14mu} e\text{-}{boundary}\text{:}}\mspace{14mu} - \psi_{0} + \frac{\pi}{2}} \leq \phi \leq {\psi_{0} + \frac{\pi}{2}}}{{{{Bottom}\mspace{14mu} e\text{-}{boundary}\text{:}}\mspace{14mu} - \psi_{0} - \frac{\pi}{2}} \leq \phi \leq {\psi_{0} - \frac{\pi}{2}}}{{{{Left}\mspace{14mu} e\text{-}{boundary}\text{:}\mspace{14mu}\psi_{0}} + \frac{\pi}{2}} \leq \phi \leq {{- \psi_{0}} + \frac{3\pi}{2}}}{{{{Right}\mspace{14mu} e\text{-}{boundary}\text{:}\mspace{14mu}\psi_{0}} - \frac{\pi}{2}} \leq \phi \leq {{- \psi_{0}} + \frac{\pi}{2}}}} & (18)\end{matrix}$Similarly there are the 4 obvious boundaries of the output image (notshown), referred to as the image boundaries, or i-boundaries:Top i-boundary:y_(f)=0,0≤x_(f)≤W₁Bottom i-boundary:y_(f)=H₁,0≤x_(f)≤W₁Left i-boundary:x_(f)=0,0≤y_(f)≤H₁Right i-boundary:x_(f)=W₁,0≤y_(f)≤H₁  (19)The i-boundaries are mapped to the corresponding e-boundaries (top totop, etc.) by finding a function that takes x or y to φ; the boundarymapping. For the top and bottom boundaries, any monotonic function of xcan be used and for the left and right any monotonic function of y. Tomaintain symmetry about the horizontal and vertical center axes (whichis common for most lenses); maps which are symmetric about the boundarycenters are preferred, specifically the i-boundary centers are mapped toe-boundary centers. An exemplary boundary map can be given by thefollowing linear functions:

$\begin{matrix}{{{{Top}\mspace{14mu}{map}\text{:}\mspace{14mu}\phi} = {{\phi_{T}\left( x_{f} \right)} = {\frac{\pi}{2} + \psi_{0} - {2\psi_{0}\frac{x_{f}}{W_{1}}}}}}{{{Bottom}\mspace{14mu}{map}\text{:}\mspace{14mu}\phi} = {{\phi_{B}\left( x_{f} \right)} = {{- \frac{\pi}{2}} - \psi_{0} + {2\psi_{0}\frac{x_{f}}{W_{1}}}}}}{{{Left}\mspace{14mu}{map}\text{:}\mspace{14mu}\phi} = {{\phi_{L}\left( y_{f} \right)} = {\frac{\pi}{2} + \psi_{0} + {2\left( {\frac{\pi}{2} - \psi_{0}} \right)\frac{y_{f}}{H_{1}}}}}}{{{Right}\mspace{14mu}{map}\text{:}\mspace{14mu}\phi} = {{\phi_{R}\left( y_{f} \right)} = {\frac{\pi}{2} - \psi_{0} - {2\left( {\frac{\pi}{2} - \psi_{0}} \right)\frac{y_{f}}{H_{1}}}}}}} & (20)\end{matrix}$Symmetry implies the following useful relations between the maps:φ_(T)(x _(f))=−φ_(B)(x _(f))φ_(L)(y _(f))=π−φ_(R)(y _(f))  (21)From the maps above, the x-y-z coordinates can be computed using:x=r_(xyz) sin θ_(max) cos φy=r_(xyz) sin θ_(max) sin φz=r_(xyz) cos θ_(max)  (22)where r_(xyz) is the distance from the center to the point on thee-boundary which, unlike the sphere, varies with the polar angles. Itcan be computed by plugging (22) into (17) to be:

$\begin{matrix}{\mspace{79mu}{{r_{xyz} = \frac{r_{x}r_{y}r_{z}}{\sqrt{D_{xyz}}}}{D_{xyz} = {{r_{y}^{2}r_{z}^{2}\cos^{2}{\phi sin}^{2}\theta_{\max}} + {r_{x}^{2}r_{z}^{2}\sin^{2}{\phi sin}^{2}\theta_{\max}} + {r_{x}^{2}r_{z}^{2}\cos^{2}\theta_{\max}}}}}} & (23)\end{matrix}$Equation (22) gives the envelope transform for the boundary 1250. FIG.12 shows the boundary mapping for a set of circular points 1260 andsquares 1270 on the output image boundary. The circles 1260 correspondto points on the left/right boundaries, and the squares 1270 correspondto points on the top/bottom boundaries.

More complicated boundary maps in place of (20) can be used to obtaindifferent transformations. Another example is to select φ to depend on xor y, such that an equal change in x or y, corresponds to equal distancemovements along the boundary curve. This requires computing thearc-length, and an example shall be provided below for the mapping ofinterior (non-boundary) points. Regardless of the map, equation (22)remains valid.

The next stage consists of extending the boundary map to all interiorpoints, hence getting the full envelope transformation. The idea is tomap image scan-lines (pixel rows) to the envelope by wrapping themaround the ellipsoid, with their end-points lying on the left and righte-boundaries. Consider a pixel row in the output image:Row at y_(f)=y_(f0),0≤x_(f)≤W₁  (24)The end-points of this row, (0, y_(f0)) and (W₁, y_(f0)), have theirmapping to the envelope already determined by the boundary mapping 1260.We label these end-points on the envelope boundary as follows:

$\begin{matrix}{\left. {{\left. \begin{matrix}{x_{L} = {r_{xyz}\sin\;\theta_{\max}\cos\;\phi_{L\; 0}}} \\{y_{L} = {r_{xyz}\sin\;\theta_{\max}\sin\;\phi_{L\; 0}}} \\{z_{L} = {r_{xyz}\cos\;\theta_{\max}}} \\{\phi_{L\; 0} = {\phi_{L}\left( y_{f\; 0} \right)}}\end{matrix} \right\}{Left}\mspace{14mu}{end}\text{-}{point}}\begin{matrix}{x_{R} = {r_{xyz}\sin\;\theta_{\max}\cos\;\phi_{R\; 0}}} \\{y_{R} = {r_{xyz}\sin\;\theta_{\max}\sin\;\phi_{R\; 0}}} \\{z_{R} = {r_{xyz}\cos\;\theta_{\max}}} \\{\phi_{R\; 0} = {\phi_{R}\left( y_{f\; 0} \right)}}\end{matrix}} \right\}{Right}\mspace{14mu}{end}\text{-}{point}} & (25)\end{matrix}$Note that (21) implies:x _(L) =−x _(R)y_(L)=y_(R)z_(L)=z_(R)  (26)From left-right symmetry about the y axis, the center of the pixel row(and for all rows) is mapped to the x=0 partial ellipse on theellipsoid; which will give a function θ(y_(f)). This map is mapping ofthe pixel column: x_(f)=W₁/2, and will be referred to as the polar map.Mapping the row centers to the x=0 curve is consistent with the boundarymap, which maps the centers of the top and bottom rows to the points(θ=θ_(max),φ=π/2) and (θ=θ_(max),φ=−π/2) respectively on the x=0 curve.As for the boundary maps, an arbitrary monotonic map can be used. Anexample linear polar map is given by:

$\begin{matrix}{\mspace{79mu}{{{{{{For}\mspace{14mu}{top}\mspace{14mu}{half}\mspace{14mu} y_{f}} \leq {\frac{H_{1}}{2}\text{:}\mspace{14mu}\phi}} = \frac{\pi}{2}},{\theta = {{\theta_{T}\left( y_{f} \right)} = {\theta_{\max}\left( {1 - \frac{2y_{f}}{H_{1}}} \right)}}}}{{{{{For}\mspace{14mu}{bottom}\mspace{14mu}{half}\mspace{14mu} y_{f}} > {\frac{H_{1}}{2}\text{:}\mspace{14mu}\phi}} = {- \frac{\pi}{2}}},{\theta = {{\theta_{B}\left( y_{f} \right)} = {\theta_{\max}\left( {1 - \frac{2y_{f}}{H_{1}}} \right)}}}}}} & (27)\end{matrix}$This map is continuous with the boundary maps described above, and mapsthe center of the image to the ellipsoid pole on the +z axis. On the y-zplane, it is convenient to define an azimuthal angle γ taken from the +yaxis, with positive towards the +z axis. This angle is related to θ by:

$\begin{matrix}{{\gamma = {\frac{\pi}{2} - {\left( {{sgn}\;\phi} \right) \times \theta}}}{{{sgn}\;\phi} \equiv \left\{ \begin{matrix}{{+ 1},} & {\phi \geq 0} \\{{- 1},} & {\phi < 0}\end{matrix} \right.}} & (28)\end{matrix}$The angle γ has a range of [π/2−θ_(max),π/2+θ_(max)]. From (22) and(28), the coordinates on the envelope for the center vertical line (i.e.the pixel row centers) is given by:x=0y=r_(yz) cos γz=r_(yz) sin γ  (29)Note that this is the x=0 curve on the ellipsoid, hence x=0. The y-zplane distance from the center r_(yz) is again not constant as for asphere; instead it is computed from (17) to be a function of γ:

$\begin{matrix}{{r_{yz} = \frac{r_{y}r_{z}}{\sqrt{D_{yz}}}}{D_{yz} = {{r_{y}^{2}\sin^{2}\gamma} + {r_{z}^{2}\cos^{2}\gamma}}}} & (30)\end{matrix}$The points on the envelope corresponding to the row centers will bedenoted as (x_(C),y_(C),z_(C)). FIG. 13 shows the polar map for a set ofpoints 1350, corresponding to center points on the output image.

The computations have thus provided three points on the envelope forevery pixel row, corresponding to the left/right end-points and thecenter:{right arrow over (x)}=(x _(L) ,y _(L) ,z _(L)){right arrow over (x)} _(R)=(x _(R) ,y _(R) ,z _(R)){right arrow over (x)} _(C)=(x _(C) ,y _(C) ,z _(C))  (31)The remaining row points can now be mapped by any well-behaved curvelying on the ellipsoid envelope interpolating these three points. Werefer to these curves as the row mapping curves or the row mapping. Awell-behaved means any curve that maintains the geometry of thepixel-row, in particular the curve should be mathematically smooth andthe totality of all the curves (for all pixel rows) should form a smoothsurface covering the ellipsoid. This is a general statement thatprovides extensive flexibility for varying the envelope transform.

Referring to FIG. 14, as a specific example, we take the curves to bedefined by the intersection of the planes 1420, uniquely determined bythe three points in (31), with the ellipsoid 1400. In this scenario, therow mapping curves will be an ellipse 1440 wrapping around theellipsoid, as seen in FIG. 14. The condition of smoothness isautomatically attained by the smoothness of the ellipses that resultfrom intersecting planes with an ellipsoid. The uniqueness of the threepoints for each row also ensures that each such plane is unique. Theexplicit determination of the curves is discussed next for this example.

The points (31) lie on the ellipsoid as well as the intersecting plane.This leads to various constraints that simplify the construction. Thenormal to the plane, {right arrow over (n)}=(n_(x),n_(y),n_(z)), can becomputed from the points (31) to be:

$\begin{matrix}{{{\overset{->}{v}}_{1} \equiv {{\overset{->}{x}}_{R} - {\overset{->}{x}}_{C}}}{{\overset{->}{v}}_{2} \equiv {{\overset{->}{x}}_{L} - {\overset{->}{x}}_{C}}}{\overset{->}{n} = \frac{{\overset{->}{v}}_{2} \times {\overset{->}{v}}_{1}}{{{\overset{->}{v}}_{2} \times {\overset{->}{v}}_{1}}}}} & (32)\end{matrix}$Using (26) in (32) it can be shown that n_(x)=0. A vector from {rightarrow over (x)}_(C) to any point (x,y,z) on the curve must beperpendicular to the normal since it lies on the intersecting plane:(x−x _(C) ,y−y _(C) ,z−z _(C))·(0,n _(y) ,n _(z))=0  (33)This leads to a relation between y and z on the curve:

$\begin{matrix}{z = {{{- \left( {y - y_{C}} \right)}\frac{n_{y}}{n_{z}}} + z_{C}}} & (34)\end{matrix}$The special case of n_(x)=0 needs to be handled separately; however inthis case the ellipse lies completely in the x-z plane and can be easilydetermined from the 2D ellipse equation and the points (31). Explicitlyone finds for (34):

$\begin{matrix}{{{z(y)} = {I\left( {y + J} \right)}}{I \equiv \frac{{r_{xyz}\cos\;\theta_{\max}} - {r_{yz}\sin\;\gamma}}{{r_{xyz}\cos\;\psi\;\sin\;\theta_{\max}} - {r_{yz}\sin\;\gamma}}}{J \equiv {{{- r_{yz}}\cos\;\gamma} + \frac{r_{yz}\sin\;\gamma}{I}}}{{\psi \equiv {{\phi_{L}\left( y_{f} \right)} - \frac{\pi}{2}}} = {\psi_{0} + {2\left( {\frac{\pi}{2} - \psi_{0}} \right)\frac{y_{f}}{H_{1}}}}}} & (35)\end{matrix}$The equation of the ellipsoid (17) can be used in turn to determine y asa function of x. Substituting (35), equation (17) becomes:

$\begin{matrix}{{\frac{x^{2}}{r_{x}^{2}} + \frac{y^{2}}{r_{y}^{2}} + \frac{\left( {I\left( {y + J} \right)} \right)^{2}}{r_{z}^{2}}} = 1} & (36)\end{matrix}$Solving (36) gives:

$\begin{matrix}{{{y(x)} = \frac{{- b_{y}} \pm \sqrt{b_{y}^{2} - {4a_{y}c_{y}}}}{2a_{y}}}{{a_{y} = {\frac{1}{r_{y}^{2}} + \frac{I^{2}}{r_{z}^{2}}}},{b_{y} = {2\frac{I^{2}J}{r_{z}^{2}}}},{c_{y} = {\frac{x^{2}}{r_{x}^{2}} + \frac{I^{2}J^{2}}{r_{z}^{2}} - 1}}}} & (37)\end{matrix}$The intersection ellipse is now completely determined by (35) and (37)in terms of x. The range of x is between the end-points of the ellipse:x_(L)≤x≤x_(R)  (38)Recall that the end-points on the ellipse are mapped to the end-pointsof the pixel row.

It remains to map 0≤x_(f)≤W₁, the pixels along a given row in theoutput, to (38). Any monotonic smooth function can be used, such as asimple linear map:

$\begin{matrix}{x = {x_{L} + {\left( {x_{R} - x_{L}} \right)\frac{x_{f}}{W_{1}}}}} & (39)\end{matrix}$Different functions lead to different local scaling along the xdirection. In one exemplary embodiment, another approach is outlinedbased on traversing equal distances on the envelope, which is similar tothe relation (12) for cylindrical projections, though the mathematics iscomplicated by the non-spherical envelope. On the intersecting plane1420, the curves 1440 are standard ellipses. Therefore the curves can beanalyzed by going to a rectangular two-dimensional coordinate system,{tilde over (x)},{tilde over (y)}, defined on this plane 1420. Inaddition, because there is no scaling involved in moving to thecoordinate system, all distances are maintained. Define the {tilde over(y)}-axis to be going through the point {right arrow over (x)}_(C) andperpendicular to the line segment through: {right arrow over(x)}_(L){right arrow over (x)}_(R), with positive direction determinedby the sign of y_(C). This is the line that sits in the ellipsoid y-zplane. Note the line segment {right arrow over (x)}_(L){right arrow over(x)}_(R) also lies in the intersection plane. The {tilde over (x)}-axiscan be chosen either (a) to be such that the ellipse origin is at thecenter, or (b) along the {right arrow over (x)}_(L){right arrow over(x)}_(R) line segment. The ellipse equation for these two cases takesthe form:

$\begin{matrix}{{{\frac{{\overset{\sim}{x}}^{2}}{{\overset{\sim}{r}}_{x}^{2}} + \frac{{\overset{\sim}{y}}^{2}}{{\overset{\sim}{r}}_{y}^{2}}} = {1\mspace{14mu}(a)}}{{\frac{{\overset{\sim}{x}}^{2}}{r_{x}^{2}} + \frac{\left( {\overset{\sim}{y} - b_{e}} \right)^{2}}{{\overset{\sim}{r}}_{y}^{2}}} = {1\mspace{14mu}(b)}}} & (40)\end{matrix}$The two differ only by a shift b_(e). Additional points can be computedon the ellipse by using the fact that in the x-y-z system, these pointsmust satisfy the ellipsoid equation. In particular, for case (b) we cancompute the intercepts with the {tilde over (y)}-axis. One of theintercepts is associated with the point {right arrow over (x)}_(C),which will be denoted y₁, and the other opposite intercept denoted y₂.Also in case (b), the intercepts along the {tilde over (x)}-axis axisare found to be −x_(R) (negative) and x_(R) (positive). In terms of theintercepts the unknown quantities in (40) can be written as:

$\begin{matrix}{{b_{e} = \frac{y_{1} + y_{2}}{2}}{{\overset{\sim}{r}}_{y} = {\frac{y_{1} - y_{2}}{2}}}{{\overset{\sim}{r}}_{x} = \sqrt{{x_{R}^{2}\left( {1 - \frac{b_{e}^{2}}{{\overset{\sim}{r}}_{y}^{2}}} \right)}^{- 1}}}} & (41)\end{matrix}$These quantities can be used to determine the arc-length of the ellipse(of the part on the ellipsoid). The arc-length s_(arc) is more easilycomputed in the coordinate system (a), and takes the form:

$\begin{matrix}{{s_{arc} = {2{\int_{{\overset{\_}{\alpha}}_{0}}^{\frac{\pi}{2}}{{{\overset{\sim}{r}}_{xy}\left( \overset{\_}{\alpha} \right)}d\overset{\_}{\alpha}}}}}{{{\overset{\sim}{r}}_{xy}\left( \overset{\_}{\alpha} \right)} = \frac{{\overset{\sim}{r}}_{x}{\overset{\sim}{r}}_{y}}{\sqrt{{{\overset{\sim}{r}}_{y}^{2}\cos^{2}\overset{\_}{\alpha}} + {{\overset{\sim}{r}}_{x}^{2}\sin^{2}\overset{\_}{\alpha}}}}}{{\overset{\_}{\alpha}}_{0} = {{- {{sgn}\left( y_{1} \right)}}{{sgn}\left( b_{e} \right)}{\arctan\left( \frac{b_{e}}{x_{R}} \right)}}}} & (42)\end{matrix}$Here α is the standard polar angle in the {tilde over (x)}-{tilde over(y)}plane. Equation (42) gives the length of the curve on the ellipsoidcorresponding to a specific pixel row. We select a mapping from x_(f) tox, such that an equal change in x_(f) corresponds to an equal traversedarc-length on the ellipse. This is achieved by finding for every x_(f)an angle α _(x) that satisfies:

$\begin{matrix}{{x_{f}\frac{s_{arc}}{W_{1}}} = {- {\int_{\pi - {\overset{\_}{\alpha}}_{0}}^{{\overset{\_}{\alpha}}_{x}}{{{\overset{\sim}{r}}_{xy}\left( \overset{\_}{\alpha} \right)}d\overset{\_}{\alpha}}}}} & (43)\end{matrix}$The negative sign and shift of the lower limit is required by theordering of x_(f), namely as x_(f) increases, α _(x) decreases. Thisprovides α _(x) as a function of x_(f), analogous to (12) for thecylindrical projection, which in turn gives the required x_(f) to x mapvia:x={tilde over (r)} _(xy)(α _(x)(x _(f)))×cos (α _(x)(x _(f)))  (44)Note that the integrals (42) and (43) can be computed numerically usingwell known techniques, such as the quadrature method. Equations (35),(37) and (44), along with the maps (20) and (27), give the finalenvelope transform, mapping the output image to a general ellipsoidenvelope.

The envelope boundary, defined by θ_(max), has been taken to beindependent of φ. However, the method disclosed is equally applicable tohaving θ_(max)(φ) depend on φ as in (6). The main difference being thatthe constant θ_(max) needs to be replaced by θ_(max)(φ), with φ firstdetermined by the boundary maps. Another subtle issue for ellipsoidenvelopes is that the envelope boundary may not lie in a plane, whereasthe row mapping curves (ellipses) are planar. As the ellipses approachthe top/bottom e-boundaries, they need to smoothly join to the boundarycurves. This can be handled by generating the ellipses from a few rowsbelow and above the top and bottom pixels rows respectively, which arethen interpolated with the boundary curves over the remaining rows. Thisis not an issue for spherical envelopes, where the envelope boundary isplanar.

The above method includes several parameters or means for customization:

-   -   I. Ellipsoid semi-axes lengths: r_(x),r_(y),r_(z)    -   II. Boundary partition angle: ψ₀    -   III. Boundary maps:        φ_(T)(x_(f)),φ_(B)(x_(f)),φ_(L)(y_(f)),φ_(R)(y_(f))    -   IV. Polar maps: θ_(T)(y_(f)),θ_(B)(y_(f))    -   V. Row mapping curves: (x_(f),y_(f),z_(f))→(x,y,z) for pixel        rows        This provides extensive flexibility for modifying the transform        to develop different types of ZCL corrections. Envelope        transforms for a 270° and 180° UWA lens are shown in FIGS. 15        and 16 for a set of points on the output image. For both cases,        the specific maps developed above ((20), (27), (35), (37) and        (44)) were used, with the parameters set as:

${{270{^\circ}\text{:}\mspace{14mu} r_{x}} = {r_{z} = 1}},{r_{y} = 2},{\psi_{0} = \frac{\pi}{6}}$${{180{^\circ}\text{:}\mspace{14mu} r_{x}} = {r_{y} = 1}},{r_{z} = 0.75},{\psi_{0} = \frac{\pi}{6}}$Both cases use an ellipsoid envelope, with the second case being closerto a spherical envelope.

The zero-content-loss correction transformation is again built bycombining the envelope mapping with the lens mapping. As mentionedbefore, computing the envelope transform requires no knowledge of thelens mapping. The only information required from the lens was its fieldof view characterization, or θ_(max), and that it be a conical-UWA lens.This implies that the same envelope transform can be used for a circularand a non-circular UWA lens; provided they have the same θ_(max) value.In other words, the method is independent of the imaging behavior of thelens. This also means that distortions introduced during themanufacturing of UWA lenses or from other sources, causing non-idealbehavior for that lens, can be accounted for provided they arecharacterized in the lens mapping equation. For the lens mappingequation (4), the zero-content-loss correction becomes:

$\begin{matrix}{{u = {\alpha\theta cos\phi}}{v = {{\alpha\theta}\;\sin\;\phi}}{{\alpha = {R_{I}\theta_{\max}^{- 1}}},{R_{I}\mspace{14mu}{radius}\mspace{14mu}{of}\mspace{14mu}{lens}\mspace{14mu}{image}}}{{{\theta\left( {x,y,z} \right)} = {\arccos\frac{z}{\sqrt{x^{2} + y^{2} + z^{2}}}}},{{\phi\left( {x,y,z} \right)} = {\arctan\left( \frac{y}{x} \right)}}}{{x\left( x_{f} \right)} = {{{\overset{\sim}{r}}_{xy}\left( {{\overset{\_}{\alpha}}_{x}\left( x_{f} \right)} \right)} \times {\cos\left( {{\overset{\_}{\alpha}}_{x}\left( x_{f} \right)} \right)}}}{{y(x)} = \frac{{- b_{y}} \pm \sqrt{b_{y}^{2} - {4a_{y}c_{y}}}}{2a_{y}}}{{z(y)} = {I\left( {y + J} \right)}}} & (45)\end{matrix}$Only the main equations are listed in (45). Applying (45), with the 180°envelope transform (see FIG. 16) settings to the input image of FIG. 6,gives the output image shown in FIG. 17. All content in the original UWAimage of FIG. 6 is visible in this correction, which further validatesthe method as generating ZCL images. The overall perspective issubstantially improved giving the impression of a standard (SLR) camerapicture. The improvements over the cylindrical projection results inFIGS. 8 to 10 are clearly seen. At the poles, perspective is recoveredvery well, with no infinite stretching. This is very apparent in thecarpet, where no tearing is seen. The curved vertical object lines inthe UWA image are almost completely straightened (as needed forperspective) without excessively curving the diagonal object lines. Thismaintains the perspective towards the left and right rooms to a muchhigher degree. The ceiling diagonals are kept diagonal. The rooms do notappear, as they did in the cylindrical projection, to lie along a line,but maintain their left right viewpoints to a much higher degree.Objects in the in the center one-third of the corrected image, such asthe central room, shelves, paintings, vents, chairs, have betterperspective in comparison with the UWA image. These improvements in thecenter, which is the most important part of the image, do not come atthe expense of losing any content. This is very critical for securityapplications, where movements in the full view need to be monitored,even though the main focus may be in the middle of the view. FIG. 18shows the ZCL image for the 270° UWA lens input image of FIG. 7. Here aspherical envelope was used with the envelope mapping shown in FIG. 19;wherein again a much improved perspective is seen, considering such awide FOV with large parts of the image coming from objects behind thecamera. The ZCL image preserves all content, straightening object linesnear the circumference without blowing up the poles, to provide an imagemuch closer to a standard camera.

By varying the envelope parameters, variations of the correction can bemade, including obtaining ZCL images which allow panoramic stitching,without “blowing up” the poles. The transformations obtained usingcylindrical projection can also be recovered with an appropriate choiceof parameters. Thus the zero-content-loss images of FIGS. 8 and 9 can beviewed as specific cases of the general ellipsoid construction above. Inparticular, to recover FIG. 9, one takes ψ₀=0. Furthermore, any UWA lenswith the same θ_(max), regardless of the captured image shape (circle,ellipse, etc.) can also be handled. FIG. 20 shows a simulated image fromUWA lens whose image appears as an ellipse. Such UWA lenses have thebenefit of covering more of the sensor, which is more commonlyrectangular than a square. This provides more input pixels containingimage data and can lead to improved quality. In this case, the lensmapping equation (2) takes the form:

$\begin{matrix}{{\phi^{\prime} = \phi}{r^{\prime} = {\alpha_{e}r_{exy}\theta}}{r_{exy} = \frac{r_{ex}r_{ey}}{\sqrt{{r_{ex}^{2}\sin^{2}\phi} + {r_{ey}^{2}\cos^{2}\phi}}}}} & (46)\end{matrix}$Using lens mapping of (46) (instead of (4)) in (45) leads to the sameimage as in FIG. 17. The ZCL framework disclosed supports all such morecomplicated lenses. As expected, there is some (residual) distortionremaining, with the perspective correction not exact, which is expectedwhen mapping UWA views to 2D images. However, the residual distortioncan be selectively eliminated in regions of interest within the ZCLframework, as outlined below.

According to an embodiment of the invention, the full view ZCLcorrection can be combined with local exact perspective corrections. Asmentioned in regard to standard non-wide angle lens cameras, the basicenvelope for a standard perspective camera is a plane encompassing theFOV, and perpendicular to the optical axis. The associated envelopetransform is a linear mapping of the 2D output image coordinates to the2D coordinates on the envelope plane. This suggests that to obtain localperspective correction for a UWA lens, along a specific directionspecified by (θ_(p1),φ_(p1)), the envelope locally needs to be a planeperpendicular to the axis through (θ_(p1),φ_(p1)). To maintain the ZCLcorrection away from the (θ_(p1),φ_(p1)) neighborhood, the envelopeshould remain an ellipsoid, with the transition being smooth.Accordingly, the method for local perspective correction consists ofsmoothly stitching the two envelopes (ellipsoid and plane) and theirtransformations together in the region of interest, and retaining theZCL transformation outside the region of interest. Let {right arrow over(F)}_(e1) and {right arrow over (F)}_(e2) be two envelopetransformations defined over (x_(f),y_(f)), mapping (x_(f),y_(f)) to(x,y,z). (For the local perspective correction formulation, we take theorigin in (x_(f),y_(f)) to be the center of the image, with positive tothe right and the top.) The full (x_(f),y_(f)) domain will be denotedA_(D). The domain is split into three disjoint sections: a region ofinterest A_(I), a transition region A_(T) surrounding A_(I), and a baseregion A_(B) containing all points not in A_(I) or A_(T). The stitchingbuilds a new envelope transform {right arrow over (F)}_(es), that isidentical to {right arrow over (F)}_(e1) on A_(B), identical to {rightarrow over (F)}_(e2) on A_(I), and interpolates between {right arrowover (F)}_(e1) and {right arrow over (F)}_(e2) on A_(T). This isdescribed by the following equation:

$\begin{matrix}{\mspace{79mu}{{{\left( {x_{f},y_{f}} \right) \in A_{D}} = {A_{B}\bigcup A_{T}\bigcup A_{I}}}\mspace{79mu}{\left( {x,y,z} \right) = {{\overset{->}{F}}_{es}\left( {x_{f},y_{f}} \right)}}{{{\overset{->}{F}}_{es}\left( {x_{f},y_{f}} \right)} = \left\{ \begin{matrix}{{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)},} & {{{if}\mspace{14mu}\left( {x_{f},y_{f}} \right)} \in A_{B}} \\{{{\overset{->}{F}}_{e\; 2}\left( {x_{f},y_{f}} \right)},} & {{{if}\mspace{14mu}\left( {x_{f},y_{f}} \right)} \in A_{I}} \\{{{{f_{1}\left( {x_{f},y_{f}} \right)}{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)}} + {{f_{2}\left( {x_{f},y_{f}} \right)}{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)}}},} & {{{if}\mspace{14mu}\left( {x_{f},y_{f}} \right)} \in A_{T}}\end{matrix} \right.}}} & (47)\end{matrix}$where f₁,f₂ are appropriate interpolating functions, which we take asscalar functions. Note since the definition of the envelop transformincludes a defining envelope, equation (47) implicitly defines the newstitched envelope. It is also useful to define the regions in the(x,y,z) space via a global transformation from (x_(f),y_(f)) to (x,y,z).If {right arrow over (G)}(x_(f),y_(f)) is such a globally defined map,the regions and stitching takes the form:

$\begin{matrix}{\mspace{79mu}{{{\left( {x,y,z} \right) \in {\overset{\_}{A}}_{D}} = {{\overset{\_}{A}}_{B}\bigcup{\overset{\_}{A}}_{T}\bigcup{\overset{\_}{A}}_{I}}}\mspace{79mu}{\left( {x,y,z} \right) = {{\overset{->}{F}}_{es}\left( {x_{f},y_{f}} \right)}}{{{\overset{->}{F}}_{es}\left( {x_{f},y_{f}} \right)} = \left\{ \begin{matrix}{{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)},} & {{{if}\mspace{14mu}{\overset{->}{G}\left( {x_{f},y_{f}} \right)}} \in {\overset{\_}{A}}_{B}} \\{{{\overset{->}{F}}_{e\; 2}\left( {x_{f},y_{f}} \right)},} & {{{if}\mspace{14mu}{\overset{->}{G}\left( {x_{f},y_{f}} \right)}} \in {\overset{\_}{A}}_{I}} \\{{{{f_{1}\left( {x_{f},y_{f}} \right)}{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)}} + {{f_{2}\left( {x_{f},y_{f}} \right)}{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)}}},} & {{{if}\mspace{14mu}{\overset{->}{G}\left( {x_{f},y_{f}} \right)}} \in {\overset{\_}{A}}_{T}}\end{matrix} \right.}}} & (48)\end{matrix}$Here the bar distinguishes the two approaches. This stitching can beviewed as moving the pixels in the region of interest to recoverperspective, which in turn twists and distorts the pixels in thetransition region, and retaining the original ZCL correction in the baseregion. Thus we push the residual distortions out of the region ofinterest to the transition region. The function {right arrow over(F)}_(e1) becomes the ZCL envelope transform (45), which is re-writtenas:

$\begin{matrix}{{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)} = {\begin{bmatrix}x \\y \\z\end{bmatrix} = \begin{bmatrix}{{{\overset{\sim}{r}}_{xy}\left( {{\overset{\_}{\alpha}}_{x}\left( x_{f} \right)} \right)} \times {\cos\left( {{\overset{\_}{\alpha}}_{x}\left( x_{f} \right)} \right)}} \\\frac{{- b_{y}} \pm \sqrt{b_{y}^{2} - {4a_{y}c_{y}}}}{2a_{y}} \\{I\left( {y + J} \right)}\end{bmatrix}}} & (49)\end{matrix}$The function {right arrow over (F)}_(e2) is taken as the perspectiveenvelope transform, which is constructed below.

To construct the perspective envelope transform, we first take the+z-axis as the optical axis (θ_(p1)=0,φ_(p1)=0) for simplifying theconstruction. In this case the envelope would be the plane at z=z_(m1),which is orthogonal to the z-axis. By varying z_(m1), the amount ofmagnification can be changed, hence the z position is kept variable.Corresponding to the direction (θ_(p1),φ_(p1)), we can associate theunique point in the output image coordinate space (x_(f),y_(f)), denoted(x_(fp1),y_(fp1)), to which that direction is mapped under the ZCLenvelope transform (45). By definition of the envelop transform thispoint exists and can be computed from (45). The point (x_(fp1),y_(fp1))is the center of the region of interest A_(I), about which localperspective correction is required. Knowing (θ_(p1),φ_(p1)),(x_(fp1),y_(fp1)) can be computed and vice-versa. With thesedefinitions, the perspective envelope transform about(θ_(p1)=0,φ_(p1)=0) can be written as:

$\begin{matrix}{{x = {s_{x}\left( {x_{f} - x_{{fp}\; 1}} \right)}}{y = {s_{y}\left( {y_{f} - y_{{fp}\; 1}} \right)}}{z = z_{m\; 1}}{s_{x} = {s_{y} = \frac{z_{m\; 1}{\tan\left( \theta_{pFOV} \right)}}{l_{p}}}}} & (50)\end{matrix}$The scale factors (s_(x),s_(y)) are shown determined by the perspectivefield of view 2′_(pFOV), and the size l_(p), of the perspectivecorrected region in (x_(f),y_(f)). The perspective envelope transformabout an arbitrary direction (θ_(p1),φ_(p1))≠(0,0), can be computed byrotating (50). To bring the (0, 0) direction to an arbitrary(θ_(p1),φ_(p1)) direction, a rotation of ±θ_(p1) degrees about thenegative y-axis (i.e. θ_(p1) degrees counter-clockwise in the xz-plane,when viewed from the negative y-axis) followed by a rotation of φ _(p1)about the positive z-axis, is used. Here φ _(p1) is the equivalent ofφ_(p1), when measured from the positive or negative x-axis such that−π/2≤φ _(p1)≤π/2. If −π/2≤φ_(p1)≤π/2, the negative sign for θ_(p1) isused, otherwise the positive. We also include an initial rotation ofφ_(0p1) about the positive z-axis, to allow rotating the perspectivecorrection in order to get better alignment (mathematical smoothness)within the surrounding zero-content-loss correction. The finalperspective envelope transformation takes the form:

$\begin{matrix}{{{{\overset{->}{F}}_{e\; 2}\left( {x_{f},y_{f}} \right)} = {\begin{bmatrix}x \\y \\z\end{bmatrix} = {{R\left( {\phi_{0p\; 1},\theta_{p\; 1},\phi_{p\; 1}} \right)}\begin{bmatrix}{s_{x}\left( {x_{f} - x_{{fp}\; 1}} \right)} \\{s_{y}\left( {y_{f} - y_{{fp}\; 1}} \right)} \\z_{m\; 1}\end{bmatrix}}}}{{R\left( {\phi_{0p\; 1},\theta_{p\; 1},\phi_{p\; 1}} \right)} = {{R_{z}\left( {\overset{\_}{\phi}}_{p\; 1} \right)}{R_{y}\left( {\pm \theta_{p\; 1}} \right)}{R_{z}\left( \phi_{0p\; 1} \right)}}}{{\overset{\_}{\phi}}_{p\; 1} = \left\{ {{\begin{matrix}{\phi_{p\; 1},} & {{{if} - {\pi/2}} \leq \phi_{p\; 1} \leq {\pi/2}} \\{{\phi_{p\; 1} - \pi},} & {{{if}\mspace{14mu}{\pi/2}} \leq \phi_{p\; 1} \leq {3{\pi/2}}}\end{matrix}{R_{y}(\theta)}} = {{\begin{bmatrix}{\cos(\theta)} & 0 & {- {\sin(\theta)}} \\0 & 1 & 0 \\{\sin(\theta)} & 0 & {\cos(\theta)}\end{bmatrix}{R_{z}(\theta)}} = \begin{bmatrix}{\cos(\theta)} & {- {\sin(\theta)}} & 0 \\{\sin(\theta)} & {\cos(\theta)} & 0 \\0 & 0 & 1\end{bmatrix}}} \right.}} & (51)\end{matrix}$The angle φ_(0p1) can be adjusted to control amount of “twisting” in thetransition region; normally a value of φ_(0p1)≈−φ_(p1) gives goodresults. FIG. 21a shows the two envelop transforms, the ZCL ellipsoid2100 and perspective plane 2150, about a direction of(θ_(p1)=65°,φ_(p1)=−35°. It remains to define the regions and computethe stitching.

For UWA lenses with a conical FOV defined by (6), it is natural to usecircular regions of interest about a given direction in the (x,y,z)space. The reason being that the (x,y,z) space gives the real worlddirection in which the lens is capturing, along which a perspective viewis sought. These circular regions are basically the intersection of acone of some angle with the perspective envelope plane. Using (48) and(50), the circular region of interest, and the transition and baseregions, can be written as:

$\begin{matrix}{{\begin{bmatrix}x \\y \\z\end{bmatrix} = {{\overset{->}{G}\left( {x_{f},y_{f}} \right)} = \begin{bmatrix}{s_{x}\left( {x_{f} - x_{{fp}\; 1}} \right)} \\{s_{y}\left( {y_{f} - y_{{fp}\; 1}} \right)} \\0\end{bmatrix}}}{r \equiv {{\overset{->}{G}\left( {x_{f},y_{f}} \right)}}}{{\overset{\_}{A}}_{I}:{r \leq r_{I}}}{{\overset{\_}{A}}_{T}:{r_{I} < r \leq r_{T}}}{{\overset{\_}{A}}_{B}:{r_{T} < r}}} & (52)\end{matrix}$The rotation in (51) ensures that these regions are transformed to thecorrect circular regions in the required direction. The interpolationcan now be done in r . As an example, a linear function combined with anadjustable exponent κ_(e) is used for the interpolation. The finalzero-content-loss envelop transform, with local perspective correctionabout (θ_(p1),φ_(p1)), or equivalently (,x_(fp1),y_(fy1)), is:

$\begin{matrix}{\begin{bmatrix}x \\y \\z\end{bmatrix} = {{{\overset{->}{F}}_{es}\left( {x_{f},y_{f}} \right)} = \left\{ {{{\begin{matrix}{{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)},} & {{{if}\mspace{14mu} r_{T}} < r} \\{{{\overset{->}{F}}_{e\; 2}\left( {x_{f},y_{f}} \right)},} & {{{if}\mspace{14mu} r} \leq r_{I}} \\{{{{f_{1}\left( {x_{f},y_{f}} \right)}{{\overset{->}{F}}_{e\; 1}\left( {x_{f},y_{f}} \right)}} + {{f_{2}\left( {x_{f},y_{f}} \right)}{{\overset{->}{F}}_{e\; 2}\left( {x_{f},y_{f}} \right)}}},} & {{{if}\mspace{14mu} r_{I}} < r \leq r_{T}}\end{matrix}\mspace{79mu}{f_{2}\left( {x_{f},y_{f}} \right)}} = \left( \frac{r_{T} - r}{r_{T} - r_{I}} \right)^{\kappa_{e}}},\mspace{79mu}{{f_{1}\left( {x_{f},y_{f}} \right)} = {1 - {f_{2}\left( {x_{f},y_{f}} \right)}}}} \right.}} & (53)\end{matrix}$The two component transforms being, {right arrow over (F)}_(e1) as in(45) and {right arrow over (F)}_(e2) as in (51). FIG. 21b shows theresulting stitched envelope mapping. Lastly the ZCL correction withlocal perspective correction is obtained by applying the lens mappingequation (4) to (53), analogous to what was done in the first fourequations of (45). About any direction, a local perspective adjustmentcan be made using the above approach. It is also possible to definemultiple regions of interest where local perspective correction isneeded. This is achieved by extending the stitching of (47) to include aunion of all disjoint regions of interest, each with its own perspectiveplanar envelope, and individually interpolated with the startingzero-content-loss envelope transform. An example of this is shown below.Hence, the method has provided a zero-content-loss correction that islocally-perspective-adaptive (LPA).

FIGS. 22 to 25 are discussed next to illustrate the zero-content-lossLPA correction. FIG. 22 is the UWA reference image (as in FIG. 6) withtwo regions of interest 2210 and 2220 within the dotted contours. Bothregions correspond to a 30° conical field of view (θ_(pFOV)=15°), withthe 2210 at direction (θ_(p1),φ_(p1))=(65°,−35°) and the 2220 at(θ_(p1),φ_(p1))=(55°,140°) in this exemplary illustration. Regions 2230and 2240 at immediate neighborhood of the regions of interest are thetransition regions. In a perspective image looking in these directions,the contours would be perfect circles. As expected the UWA greatlydistorts the circles into ellipse-like contours. FIG. 23 shows thereference image after applying the ZCL correction, identical to FIG. 17,but now with the regions of interest 2310 and 2320 highlighted. The fullimage perspective is largely improved, as previously discussed; howeverdistortion still remains as seen in the curvature of the straight objectlines. With the contours drawn, the residual distortion becomes clear.In a locally perspective correct view the contours would be circles,rather than the egg-shaped curves seen. Next the zero-content-loss LPAcorrection is applied to locally correct perspective for the region ofinterest 2310. FIG. 24 shows the results using a z_(m1) value of 1; thisbeing slightly larger than the r_(z)=0.75 value will provide a slightlocal zoom. The perspective is now fully corrected locally, with thecontours being circles and all object lines straight within the circle.Note how the transition region 2430 has now become distorted, as if theresidual distortion has been pushed out into the transition region. Awayfrom the transition region 2430, the ZCL correction is untouched. Theimage within the circle 2430 is slightly rotated in order to smoothlyconnect with the surrounding ZCL correction. This was achieved by usingφ_(0p1)=25°; by varying this angle the rotation can be changed and inturn the twisting in the transition region. The rotation does notdistort the perspective correction. The image in 2410, being perspectivecorrect, can now be analyzed for features and other video analyticprocessing, just like a standard camera image (this is discussed furtherbelow). At the same time being a ZCL correction, the entire scene isstill available for viewing and analysis. FIG. 25 shows a similarcorrection applied to both regions of interest 2510 and 2520.Perspective correction is now locally achieved in both circles.Following the same procedure, multiple regions can also be corrected.Further, by varying the size of the transition region, the spreading ofthe distortion can be controlled.

The above examples have worked with small circular regions of interest.It is also possible to locally perspective correct large regions and usenon-circular regions of interest. A useful non-circular geometry isdefining rectangular regions on the output image, in which perspectivecorrection is needed, as follows:(x _(f) ,y _(f))ϵA _(D) =A _(B) ∪A _(T) ∪A _(I)A _(I)={(x _(f) ,y _(f)) such that |x _(f) −x _(fp1) |≤l _(Ix) ,|y _(f)−y _(fp1) |≤l _(Iy)}A _(T)={(x _(f) ,y _(f)) such that l _(Ix) <|x _(f) −x _(fp1) |≤l _(Tx),l _(Iy) <|y _(f) −y _(fp1) |≤l _(Ty)}A _(B)={(x _(f) ,y _(f)) such that l _(Tx) <|x _(f) −x _(fp1) |,l _(Ty)<|y _(f) −y _(fp1)|}  (54)Here {l_(Ix),l_(Iy),l_(Tx),l_(Ty)} define the sizes of the rectangularregions. FIG. 26a shows a rectangular region based zero-content-loss LPAcorrection for a wide field of view (θ_(pFOV)=45°) about the z-axisdirection (directly in front of the lens). The full perspectivecorrection of the front view within the zero-content-loss correction isseen. FIG. 26b shows the front view correction with a zoom applied. FIG.26c shows a similar correction of the combined right view (direction(θ_(p1),φ_(p1))=(45°,0°)) and left view (direction(θ_(p1),φ_(p1))=(45°,180°)); here the distortions are pushed into themiddle room and to the floor and ceiling. Lastly, FIG. 26d combinescorrections for the left, center and right views, giving threeperspective corrected regions.

It should be noted that the above corrections being zero-content-loss,include all content from the original scene (FIG. 22). According to oneembodiment, it is possible to extract only the regions of interest fromthe ZCL LPA correction, eliminating all other pixels in the transitionand base regions, to present a set of perspective corrected views. FIG.27 illustrates an example of such extraction wherein four roomssurrounding a 360° area (captured by e.g. two fisheyes) are individuallycorrected for perspective and displayed independently. Moreover, onewould still be able to select smaller areas of interest in each view forfurther processing as disclosed before. Clearly, in this type ofapplications the discarded pixel information is not of great interest.An example is a video-conference setting where the faces of participantsare the most important features.

According to another embodiment, the disclosed method is used tofacilitate optional follow-up video content analysis (VCA), also knownas video analytics. In order to perform object and pattern recognitionin an image, or generally to analyze a video automatically; good inputimages are required. That means all sorts of distortions must beminimized. For instance, a video monitoring camera can be equipped withone or more of motion sensors light/sound sensors, etc. that can triggerrecording events for a period of time. In that case, the content of therecorded video will be needed to recognize the moving object such as theface of a person as illustrated in FIG. 28. The embodied methods can beused to select areas of interest 2810 and 2820 around the moving object,and correct the perspective while other contents of the video and theirrelation to the object of interest are preserved. FIG. 29 shows how theareas of interest in the vicinity of the faces 2810 and 2820 arecorrected. In this example, the faces are further zoomed in (and couldbe further rotated) in order to aid an image recognition means. At thesame time, being a zero-content-loss map, all scene information ismaintained for additional detection or analytics in parallel. Thisadaptive correction not only is applicable to a plurality of zoneswithin the FOV, but can be applied also to multiple frames of a videowith different time stamps. Accordingly, FIG. 29 can also be interpretedas one person at various locations under surveillance at differenttimes.

In one preferred embodiment, the methods disclosed above are implementedin an electronic processor. The processor can be integrated to an imagecapture, image display or an image processing device. The disclosedmethods can also be implemented as software for use in acomputer-readable medium.

While the above description provides examples of the embodiments, itwill be appreciated that some features and/or functions of the describedembodiments are susceptible to modification without departing from thespirit and principles of operation of the described embodiments.Accordingly, what has been described above has been intended to beillustrative of the invention and non-limiting and it will be understoodby persons skilled in the art that other variants and modifications maybe made without departing from the scope of the invention as defined inthe claims appended hereto.

The invention claimed is:
 1. A method for transforming a plurality ofinput video images captured from a scene through at least one ultrawide-angle (UWA) lens using at least one electronic processor, whereinthe at least one UWA lens is characterized by a corresponding lenstransformation that maps three dimensional (3D) object space onto aplane that displays the plurality of input video images, said methodcomprising: (a) obtaining the plurality of input video images datacaptured through the at least one UWA lens; (b) constructing a twodimensional (2D) surface envelope in accordance with one or more areasof the scene selected for perspective correction, wherein the 2D surfaceenvelope fully encompasses field of view of the at least one UWA lens;(c) constructing an envelope transformation, independently from thecorresponding UWA lens transformation, that maps an output image planeonto the 2D surface envelope uniquely for every point in the outputimage plane; (d) concatenating the corresponding UWA lens transformationfor the at least one UWA lens and the envelope transformation to obtaina zero-content-loss transformation, that provides a content preservingmapping between the input and output image planes, and wherein the UWAlens transformation is independent of image content; and (e) applyingthe zero-content-loss transformation to the plurality of input videoimages data to obtain a plurality of output video images, wherein imageperspective is accordingly transformed for the one or more selectedareas of the scene; wherein at least one of the surface envelope and theenvelope transformation is modified to locally adjust for perspectivecorrections in at least one region of interest; and wherein the envelopetransformation is modifiable without change to the surface envelope. 2.The method of claim 1, wherein the at least one region of interest isselected manually.
 3. The method of claim 1, wherein the at least oneregion of interest is selected automatically in response to changes inthe scene.
 4. The method of claim 1, wherein at least one of scaling,zoom, rotation and reflection transformation is applied to the at leastone perspective corrected region of interest.
 5. The method of claim 4,wherein the at least one corrected region of interest is supplied forvideo content analysis.
 6. The method of claim 1, wherein the localperspective correction at a region of interest in a direction from thelens is obtained by: (i) constructing a plane envelope perpendicular toan axis connecting the lens to center of the region of interest; (ii)smoothly stitching the plane envelope and the surface envelope, andtheir transformations in the region of interest.
 7. The method of claim6, wherein local magnification is achieved by moving the plane envelopealong the axis connecting the lens to the region of interest.
 8. Themethod of claim 1, wherein the at least one UWA lens covers up to a 360°field of view.
 9. The method of claim 1, wherein the lens map isprovided by the lens manufacturer.
 10. The method of claim 1, whereinthe lens map is mathematically modeled.
 11. The method of claim 1,wherein the lens map is empirically characterized.
 12. The method ofclaim 1, wherein the surface envelope and the envelope transformationare constructed in a parameterized form.
 13. The method of claim 1,wherein a cylindrical projection is used to map the output image planeto the surface envelope.
 14. The method of claim 1, wherein mapping theoutput image plane to the surface envelope is represented as a function.15. The method of claim 1, wherein the plurality of input video imagesis circular and the envelope is selected as a partial surface of asphere, having a boundary at the field of view of the lens.
 16. Themethod of claim 1, wherein the plurality of input video images isnon-circular and the envelope is selected as a partial surface of anellipsoid, having a boundary at a maximum field of view of the lens. 17.The method of claim 15, wherein the envelope transformation maps anoutput rectangular image comprising a boundary and a plurality of pixelrows, said transformation is constructed by: (i) mapping pixels on theoutput image boundary to the boundary of the ellipsoid envelope; (ii)mapping centers of the plurality of pixel rows to an ellipse on theellipsoid; and (iii) mapping non-boundary pixels for each of theplurality of pixel rows to a curve on the ellipsoid, said curveconstructed by interpolating a plane containing map of the center pixeland maps of corresponding two boundary pixels.
 18. The method of claim1, wherein the zero-content-loss output image is displayed in itsentirety.
 19. The method of claim 1, wherein a panoramic view isextracted from the zero-content-loss output image for display.
 20. Themethod of claim 1, wherein a plurality of perspective corrected regionsis extracted from the output image to display independently.
 21. Themethod of claim 20, wherein at least one of scaling, zoom, rotation andreflection transformation is applied to the perspective correctedregions.
 22. An electronic processor for transforming a plurality ofinput video images captured from a scene through at least one ultrawide-angle (UWA) lens, wherein each of the at least one UWA lens ischaracterized by a corresponding lens transformation that maps threedimensional (3D) object space onto a plane that displays the pluralityof input video images, said processor comprising: (a) means to obtainthe plurality of input video images data captured through the at leastone UWA lens; (b) means to select a two dimensional (2D) surfaceenvelope in accordance with one or more areas of the scene selected forperspective correction, wherein the 2D surface envelope fullyencompasses field of view of the at least one UWA lens; (c) means toconstruct an envelope transformation, independently from thecorresponding UWA lens transformation, that maps an output image planeonto the 2D surface envelope uniquely for every point in the outputimage plane; (d) means to concatenate the corresponding UWA lenstransformation for the at least one UWA lens and the envelopetransformation to obtain a zero-content-loss transformation thatprovides a content preserving mapping between the input and output imageplanes, and wherein the UWA lens transformation is independent of imagecontent; and (e) means to apply the zero-content-loss transformation tothe plurality of input video images data to obtain a plurality of outputvideo images, wherein image perspective is accordingly transformed forthe one or more selected areas of the scene; wherein said processorfurther comprises means to modify at least one of the surface envelopeand the envelope transformation to locally adjust for perspectivecorrections in at least one region of interest; and wherein the envelopetransformation is modifiable without change to the surface envelope. 23.The processor of claim 22, wherein said processor further comprisesmeans to enable manual selection of the at least one region of interest.24. The processor of claim 22, wherein said processor further comprisesmeans to enable automatic selection of the at least one region ofinterest in response to changes in the scene.
 25. The processor of claim22, wherein said processor further comprises means to apply at least oneof scaling, zoom, rotation and reflection transformation to the at leastone perspective corrected region of interest.